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tj.  AUTRACT  iOO  »«ardU 

Patterns  of  40  to  80  Hz  oscillation  have  been  observed  by  researchers  of  this 
laboratory  in  the  large  scale  activity  not  only  of  olfactory  cortex,  but  also 
visual  neocortex,  and  shown  to  predict  the  olfactory  and  visual  pattern  recognition 
responses  of  a  trained  animal.  Similar  observations  of  40  Hz  oscillation  in  auditory 
and  motor  cortex,  and  in  the  retina  and  EMC  have  been  reported.  It  thus  appears 
that. cortical  computaiton  in  general  may  occur  by  dynamical  interaction  of  resonant 
modes’,  as  we  have  long  thought  to  be  the  case  in  the  olfactory  system.  The 
oscillation  can  serve  a  macroscopic  clocking  function  and  entrain  or  "bind"  the 
relevant  microscopic  activity  of  disparate  cortical  regions  into  a  well  defined 
phase  coherent  collective  state  of  "gestalt".  This  can  overide  irrelevant  micro¬ 
scopic  activity  and  produce  coordinated  motor  output.  We  have  further  evidence  that 
the  oscillatory  activity  is  roughly  periodic,  but  actually  appears  to  be  chaotic 
(nonperiodic)  when  examined  in  detail.  If  this  view  is  correct,  then  networks  with 
oscillatory  and  possibly  chaotic  activity  form  the  actual  cortical  substrate  of  the 
diverse  sensory,  motor,  and  cognitive  operations  now  studied  in  static  networks.  It 
must  then  be  shown  how  those  functions  can  be  accomplished  with  oscillatory  and 
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chaotic  dynamics.  It  is  our  expectation  that  nature  makes  good  use  of  this 
dynamical  complexity and  our  intent  has  been  to  search  here  for  novel  design 
principles  that  may  underly  the  superior  performance  of  biological  systems  in 
pattern  recognition.  These  may  then  be  applied  in  artificial  systems  to 
engineering  problems. 
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1  Introduction 

Patterns  of  40  to  80  Hz  oscillation  have  been  observed  by  researchers  of 
this  laboratory  in  the  large  scale  activity  not  only  of  olfactory  cortex,  but 
also  visual  neocortex,  and  shown  to  predict  the  olfactory  and  visual  pattern 
recognition  'sponses  of  a  trained  animal.  Similar  observations  of  40  Hz 
oscillation  .  r  auditory  and  motor  cortex,  and  in  the  retina  and  EMG  have 
been  reported. 

It  thus  appears  that  cortical  computation  in  general  may  occur  by  dy¬ 
namical  interaction  of  resonant  modes,  as  we  have  long  thought  to  be  the 
case  in  the  olfactory  system.  The  oscillation  can  serve  a  macroscopic  clocking 
function  and  entrain  or  “bind”  the  relevant  microscopic  activity  of  disparate 
cortical  regions  into  a  well  defined  phase  coherent  collective  state  or  “gestalt”. 
This  can  overide  irrelevant  microscopic  activity  and  produce  coordinated  mo¬ 
tor  output.  We  have  further  evidence  that  the  oscillatory  activity  is  roughly 
periodic,  but  actually  appears  to  be  chaotic  (nonperiodic)  when  examined  in 
detail. 

If  this  view  is  correct,  then  networks  with  oscillatory  and  possibly  chaotic 
activity  form  the  actual  cortical  substrate  of  the  diverse  sensory,  motor,  and 
cognitive  operations  now  studied  in  static  networks.  It  must  then  be  shown 
how  those  functions  can  be  accomplished  with  oscillatory  and  chaotic  dy¬ 
namics.  It  is  our  expectation  that  nature  makes  good  use  of  this  dynamical 
complexity,  and  our  intent  has  been  to  search  here  for  novel  design  principles 
that  may  underly  the  superior  performance  of  biological  systems  in  pattern 
recognition.  These  may  then  be  applied  in  artificial  systems  to  engineering 
problems. 
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The  focus  of  our  work  was  to  characterize  mathematically  the  essential 
mechanisms  and  principles  of  operation  of  the  mamalian  olfactory  neural 
network  and  evaluate  its  computation  and  pattern  recognition  capabilities. 
In  this  endeavor  we  have  indeed  discovered  novel  and  powerful  network  ar¬ 
chitectures  and  learning  algorithms. 

Over  the  three  and  a  half  years  of  the  grant,  we  pubhshed  13  papers, 
and  gavt  presentations  at  16  conferences.  Network  algorithms  and  archi¬ 
tectures  'Or  pattern  recognition  have  been  constructed  from  neural  models 
of  the  olfactory  system,  using  the  mathematical  tools  of  dynamicl  systems 
theory.  These  span  a  range  from  highly  abstract  to  physiologically  detailed, 
and  employ  the  kind  of  dynamical  complexity  observed  in  olfactory  cortex, 
ranging  from  oscillation  to  chaos. 


2  Theoretical  Work 

A  new  class  of  learning  algorithms  for  the  storage  of  static  and  periodic 
attractors  in  recurrent  analog  neural  networks  was  developed.  Over  the  grant 
period,  the  original  projection  algorithm  has  been  extended  and  refined  in 
many  ways.  The  seminal  mathematical  insight  of  the  algorithm  was  that 
techniques  of  bifurcation  theory  for  analysis  of  nonlinear  dynamiced  systems 
could  be  inverted  to  produce  precise  techniques  for  synthesis  of  dynamical 
neural  networks  like  that  of  the  olfactory  system. 

A  key  feature  of  a  net  constructed  by  this  eJgorithm  is  that  the  underlying 
dynamics  is  explicitly  isomorphic  to  any  of  a  class  of  standard,  well  under¬ 
stood  nonlinear  dynamical  systems  -  a  “normal  form”.  This  system  is  chosen 
in  advance,  independent  of  both  the  patterns  to  be  stored  and  the  learning 
algorithm  to  be  used.  This  control  over  the  dynamics  permits  the  design 
of  important  a.spects  of  the  network  dynamics  independent  of  the  particular 
patterns  to  be  stored.  Stability,  basin  geometry,  and  rates  of  convergence  to 
attractors  can  be  programmed  in  the  standard  dynamical  system. 

We  were  able  to  derive  learning  algorithms  for  content  addressable  mem¬ 
ory  storage  of  static  and  oscillatory  attractors  with  analytically  guaranteed 
storage  capacity  of  N  static  and  N/2  oscillatory  attractors  in  an  N  node 
network.  This  is  in  contrast  to  the  ESTIMATES  of  4  N/  log  N  static  at¬ 
tractors  for  an  AVERAGE  Hopfield  network  (nothing  precise  can  be  said  for 
any  individual  Hopfield  network).  This  wa.s  done  with  a  simple  learning  rule 
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-  a  formula  requiring  a  matrix  inversion  -  for  first  and  third  order  weights 
in  a  recurrent  network  architecture.  There  are  no  spurious  attractors,  and 
there  is  a  Liapiinov  function  in  a  special  coordinate  system  which  governs 
the  approach  of  transient  states  to  stored  patterns. 

Later  work  extended  these  results  to  allow  guaranteed  storage  of  mul¬ 
tiple  periodic  spatiotemporal  sequences  of  up  to  N/2  total  Fourier  compo¬ 
nents.  We  showed  also  that  for  orthogonal  patterns,  the  projection  learning 
rule  reduced  to  a  Hebbian  outer  product  rule  that  allowed  local  incremental 
learning.  In  this  last  year  we  have  shown  that  N /3  chaotic  attractors  may  be 
stored  in  an  N  node  network,  and  constructed  networks  with  multiple  static, 
periodic,  and  chaotic  attractors  all  in  the  same  network. 

A  new  architecture  called  the  projection  network  was  developed  that  did 
not  require  the  higher  order  weights  (only  3  X  N  squared  weights  instead 
of  N  fourth)  yet  retained  all  the  mathematically  demonstrable  capabilities 
of  the  previous  architecture.  This  net  was  thus  more  suited  to  engineering 
applications.  Unsupervised  or  supervised  incremental  learning  algorithms  for 
pattern  classification,  such  as  competitive  learning  or  bootstrap  Widrow-Hoff 
can  easily  be  implemented  in  this  architecture. 

Network  performance  was  demonstrated  by  application  to  the  problem 
of  real  time  handwritten  digit  recognition.  An  effective  system  with  on  line 
learning  was  written  for  the  Macintosh.  It  utilizes  static,  oscillatory,  and/or 
chaotic  attractors  of  three  kinds  -  Lorenz  attractors,  Roessler  attractors,  or 
attractors  resulting  from  chaotically  interacting  oscillatory  modes,  which  we 
call  Ruelle-Takens  attractors.  This  appears  to  be  the  first  system  which  can 
use  exclusively  chaotic  dynamics  to  accomplish  pattern  recognition. 

On  the  more  scientific  front  of  biological  modeling,  the  higher  order  net¬ 
work  was  refined  into  a  biologically  minimal  network  model  of  olfactory  cor¬ 
tex  with  explicit  excitatory  and  inhibitory  neurons.  This  model  assumed  the 
least  of  known  biological  connections,  and  was  shown  to  have  the  associa¬ 
tive  memory  capacity  of  the  systems  above.  We  showed  also  that  only  N 
squared  of  the  higher  order  weights  between  excitatory  neurons  were  reqired 
for  approximate  pattern  storage  in  this  system. 
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3  An  Industrial  Application 

In  a  parallel  development  by  Freeman  and  Yao,  a  physiologically  detailed 
olfactory  system  model  wa;s  developed  in  software  and  hardware  versions.  Its 
features  were  derived  from  biologicaJ  studies  and  bring  the  model  closely  into 
conformance  with  biological  constraints.  This  model,  which  we  call  a  Kill 
set,  wtis  tested  in  respect  to  artificially  generated  data  set  and  then  with  a 
data  set  derived  from  an  industrial  source.  The  Kill  model  was  "instructed” 
with  examples  from  groups  of  objects  to  be  classified,  with  use  of  a  form  of 
the  Hebb  rule  to  modify  the  connection  weights.  The  interconnections  were 
designed  in  accordance  with  the  time  multiplexing  operation  that  enables 
connections  to  be  made  in  large  arrays  of  N  coupled  oscillators  with  2-N 
rather  than  N-square  connections. 

The  performance  of  the  Kill  model  in  classification  of  the  members  of 
this  data  set  was  compared  with  the  performance  of  a  Bayesian  statistical 
approach,  the  Hopfield  model,  and  a  back-propagation  model.  Of  the  three 
classes  of  objects,  those  having  a  relatively  high  degree  of  symmetry  were 
classified  equally  well  by  all  of  the  approaches.  The  most  difficult  set  of 
objects,  which  had  a  high  degree  of  asymmetry  and  therefore  a  large  amount 
of  extraneous  variance,  were  successfuDy  classified  only  by  the  Kill  model 
operating  in  a  chaotic  domain  and  not  in  a  limit  cycle  domain.  The  results 
indicate  that  exploration  of  chaotic  dynamics  may  be  crucially  important  for 
the  preprocessing  of  complex  high-dimensional  data  sets  prior  to  attempts 
at  classification. 
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First  Annual  INNS  Meeting,  Boston,  Mass.,  September 

Neural  Information  Processing  Systems  -  Natural  and  Synthetic,  Denver,  Colo.,  November 

1989 

U.C.  San  Diego,  Cognitive  Science  Dept.,  January 

Chairman  of  session.  Workshop  on  Neural  Networks  as  Dynamical  Systems,  U.C.,  San  Diego,  Ca.,  February 

Emergent  Computation  -  Self-Organizing,  Collective,  and  Cooperative  Phenomena  in  Natural  and  Artificial 
Computing  Networks,  Center  for  Nonlinear  Science,  Los  Alamos,  N.M.,  May 

International  Joint  Conference  on  Neural  Networks,  Wash.  D.C.,  June 

Neural  Information  Processing  Systems  -  Natural  and  Synthetic,  Denver,  Colo.,  November 

1990 

Stanford  University,  Chemistry  Dept.,  Palo  Alto,  Ca.,  March 

Lawrence  Livermore  National  Laboratory,  Institute  for  Scientific  Computing,  Livermore,  Ca.,  April 
Chairman  of  session,  International  Joint  Conference  on  Neural  Networks,  San  Diego,  Ca.,  June 
Analysis  and  Modeling  of  Neural  Systems  1,  U.C.  Berkeley,  Ca.,  July 
U.C.  Santa  Cruz,  Mathematics  Dept.,  September 

Neural  Information  Processing  Systems  -  Natural  and  Synthetic,  Denver,  Colo.,  November 

1991 

Plenary  talk.  Applications  of  Artificial  Intelligence  and  Neural  Networks,  SPIE,  Orlando,  Fla.,  April 

Stanford  University,  Artificial  Networks  Group,  Palo  Alto,  Ca.,  April 

Invited  Speaker,  Workshop  on  Complex  Dynamics  in  Neural  Networks,  Vietri,  Italy,  June 

*talk  or  poster  given  at  all  conferences  listed 
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